Stochastic Shortest Path Games and Q-Learning

We consider a class of two-player zero-sum stochastic games with finite state and compact control spaces, which we call stochastic shortest path (SSP) games. They are total cost stochastic dynamic games that have a cost-free termination state. Based on their close connection to singleplayer SSP problems, we introduce model conditions that characterize a general subclass of these games that have strong properties: the value function exists and is the unique solution of the Bellman equation, both players have optimal policies that are stationary deterministic, and the value iteration algorithm, as well as the policy iteration algorithm starting with certain wellbehaved policies, converge. We then consider the classical Q-learning algorithm that computes the value function for finite state and control SSP games that satisfy our model conditions. Q-learning is a model-free, asynchronous stochastic iterative algorithm, and by the theory of stochastic approximation involving monotone nonexpansive mappings, it is known to converge when the Bellman equation has a unique solution and its iterates are bounded with probability one. We prove the boundedness of the Q-learning iterates and thereby establish completely the convergence of Q-learning for our broad class of SSP game models. Dec 2011